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Analysis  of  Team  Communications  to  Understand  Cognitive  Processes 

used  during  Team  Collaboration 

Susan  G.  Hutchins  and  Tony  Kendall 


Macrocognition  describes  the  way  cognition  occurs  in  naturalistic,  or  real-world, 
decision-making  events  and  comprises  the  mental  activities  that  must  be  successfully 
accomplished  to  perform  a  task  or  achieve  a  goal.  The  emphasis  in  macrocognition  is 
on  the  cognitive  functions  performed  during  collaborative  team  problem  solving  and 
how  teams  can  perfonn  them.  For  this  research  we  employed  an  empirical  process  for 
evaluating  a  model  of  team  collaboration  by  analyzing  and  coding  transcripts  or  chat 
logs  that  transpired  during  several  real-world  problem-solving  events.  Team 
communications  that  transpired  when  teams  collaborated  to  solve  complex  problem¬ 
solving  tasks  from  two  decision-making  domains  were  analyzed  and  coded  using  the 
definitions  of  the  macrocognitive  processes  in  the  model  of  team  collaboration.  Two 
coders  coded  each  set  of  data  and  then  reviewed  their  coding  with  one  of  the  authors 
and  discussed  any  differences  in  interpreting  the  definitions.  Kappa  Cohen,  an  inter¬ 
rater  reliability  score,  indicated  high  levels  of  agreement  between  coders  for  both  sets 
of  data.  Consistent  with  our  previous  research  we  found  that  problem  solving  by  these 
teams  comprised  a  series  of  decisions.  That  is,  decisions  were  made  iteratively 
throughout  the  entire  problem  situation  as  opposed  to  problem  solving  culminating  in 
team  members  agreeing  on  one  big  final  decision. 


Macrocognition  is  an  emerging  field  within  the  area  of  cognitive  engineering  that  describes  the 
way  cognition  occurs  in  naturalistic,  or  real-world,  decision-making  events  (Cacciabue  & 
Hollnagell,  1995).  When  studying  macrocognition,  the  focus  is  on  the  mental  activities  that  must 
be  successfully  accomplished  to  perfonn  a  task  or  achieve  a  goal  (Klein,  Ross,  Moon,  Klein, 
Hoffman,  &  Hollnagel,  2003).  These  cognitive  functions  are  generally  performed  during 
collaborative  team  problem  solving,  where  the  emphasis  is  on  building  new  knowledge. 
Macrocognition  is  differentiated  from  microcognition  in  several  ways.  Microcognition  places  an 
emphasis  on  experimental  control  of  tasks  and  theoretical  accounts  of  specific  phenomena  while 
macrocognition  emphasizes  cognition  and  perfonnance  under  actual  working  conditions. 
Macrocognitive  phenomena  generally  occur  over  longer  time  periods,  involve  unstructured  tasks, 
and  do  not  focus  on  the  “basic”  cognitive  functions  of  microcognition  (e.g.,  attention  and 
memory). 

Macrocognition  encompasses  cognitive  processes  involved  in  detecting  problems,  developing 
and  sharing  situation  awareness,  generating  options,  using  analogues,  mentally  simulating 
courses  of  action,  planning  and  re -planning,  maintaining  vigilance,  and  assessing  risk  (Klein, 
2001).  Several  groups  of  researchers  maintain  that  research  on  macrocognition  is  needed  to 
better  understand  the  cognitive  functions  employed  by  teams  when  they  collaborate  to  solve 
challenging,  infonnation-rich,  time-compressed  problems  (Klein  et  al,  2003;  Fiore,  Smith  & 
Letsky,  2008;  Letsky  &  Warner,  2008).  This  understanding  can  then  be  applied  to  improve  the 
cognitive  engineering  of  future  systems. 
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One  goal  for  studying  macrocognition  is  to  understand  the  complexity  entailed  in  inter-  and 
intra-individual  cognition.  For  the  research  reported  in  this  paper,  we  focus  on  cognition  in 
problem-solving  teams  who  collaborate  while  performing  real-world  tasks,  in  line  with  the  view 
of  macrocognition  that  seeks  to  describe  cognitive  work  as  it  naturally  occurs  (Klein  et  al,  2003). 
We  seek  to  better  understand  how  cognition  emerges  in  problem-solving  teams  engaging  in  tasks 
involving  short-term  situations  which  require  rapid  action  to  be  taken  towards  specific  mission 
goals.  Macrocognition  is  defined,  for  this  research,  as  the  internalized  and  externalized  high- 
level  mental  processes  employed  by  teams  to  create  new  knowledge  during  complex,  one-of-a 
kind,  collaborative  problem  solving  (Burke,  2007;  Letsky,  Warner,  Fiore,  Rosen,  Salas,  2007). 
High-level  mental  processes  refer  to  the  cognitive  processes  involved  in  combining,  visualizing, 
and  aggregating  infonnation  to  resolve  ambiguity  in  support  of  the  discovery  of  new  knowledge 
and  relationships. 

Internalized  processes,  that  is,  processes  occurring  inside  the  head,  are  those  higher-level  mental 
[cognitive]  processes  that  are  not  expressed  externally  (e.g.,  writing,  speaking,  gesture),  and  can 
only  be  measured  indirectly  via  qualitative  metrics  (e.g.,  questionnaires,  cognitive  mapping, 
think  aloud  protocols,  multi-dimensional  scaling,  etc.),  or  surrogate  quantitative  metrics  (e.g., 
pupil  size,  galvanic  skin  response).  These  processes  can  become  either  fully  or  partially 
externalized  when  they  are  expressed  in  a  fonn  that  relates  to  other  individual’s  reference/ 
interpretation  systems  (e.g.,  language,  icons,  gestures,  boundary  objects  (Letsky  et  al.,  2007). 

Externalized  processes  (processes  occurring  outside  the  head)  are  those  higher-level  mental 
[cognitive]  processes  that  occur  at  the  individual  or  team  level,  and  which  are  associated  only 
with  actions  that  are  observable  and  measureable  in  a  consistent,  reliable,  repeatable  manner,  or 
through  the  conventions  of  the  subject  domain  have  standardized  meanings  (Letsky  et  al,  2007). 

The  framework  of  collaborative  problem  solving  developed  as  part  of  the  Office  of  Naval 
Research  Collaboration  and  Knowledge  Integration  (CKI)  Program  (Letsky,  Warner,  Fiore,  & 
Salas,  2007;  Warner,  Letsky,  &  Cowen,  2005)  provides  the  conceptual  foundation  for  this 
research.  The  emphasis  on  macrocognition  in  teams  was  initiated  as  part  of  a  larger  issue  of  how 
to  understand  and  facilitate  complex,  collaborative  activity  -  specifically  in  quick-response  ad 
hoc  teams.  Both  commercial  and  military  communities  are  evolving  in  response  to  an  increased 
reliance  on  socio-technical  systems,  globalization,  and  ubiquitous  infonnation  accessibility;  a 
development  which  leads  to  changes  in  the  dynamics  of  team  activity  (Letsky  &  Warner,  2008). 
The  CKI  program  seeks  to  develop  a  better  understanding  of  internalized,  non-quantifiable, 
mental  processes  at  work  as  teams  collect,  filter,  process  and  share  infonnation  for  problem¬ 
solving  purposes. 

The  objective  of  the  CKI  program  is  to  respond  to  emerging  needs  in  both  the  military  and 
business  environments  to  better  understand  and  improve  the  effectiveness  of  team  decision 
making  in  complex,  data-rich  situations.  As  part  of  this  effort,  a  model  of  team  collaboration  was 
developed  that  emphasizes  the  cognitive  aspects  of  team  collaboration  and  includes  the  major 
human  decision-making  processes  used  during  team  collaboration  (Warner,  Letsky  &  Cowen, 
2005).  The  long-range  program  objective  is  to  develop  cognitive  science-based  tools,  models, 
computational  methods,  and  human-agent  interfaces  to  help  attain  common  situation  awareness 
among  distributed  team  members,  engaged  in  asynchronous,  quick-response  collaboration  for 
issue  resolution,  or  decision  making. 
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The  goal  for  the  research  reported  here  is  to  understand  the  role  of  cognition  in  teams  who  are 
collaborating  to  solve  challenging,  dynamic,  experienced-based,  knowledge  intensive, 
ambiguous  problems.  Our  objectives  are  (1)  to  empirically  evaluate  a  model  of  team 
collaboration  (Letsky  et  ah,  2007)  by  analyzing  the  macrocognitive  processes  used  by  teams 
during  real-world  complex  decision-making  events,  (2)  to  refine  the  model  as  necessary,  and  (3) 
as  a  result  of  our  analysis  to  develop  a  better  understanding  of  the  cognitive  processes  involved 
in  team  collaboration. 

Team  Collaboration 

Many  definitions  of  collaboration  are  found  in  the  different  bodies  of  research  literature  (see 
Wood  and  Gray,  1991).  At  the  most  fundamental  level,  collaboration  refers  to  the  joint  effort  of 
two  or  more  agents  to  achieve  a  common  goal  where  collaborating  members  construct  judgments 
and  then  act  on  these  judgments  (Nosek,  2004).  A  definition  more  aligned  with  the  research 
reported  here  states  that  collaboration  occurs  “when  a  group  of  autonomous  stakeholders  of  a 
problem  domain  engage  in  an  interactive  process,  using  shared  rules,  norms,  and  structures,  to 
act  or  decide  on  issues  related  to  that  domain”  (Wood  &  Gray,  1991,  p.l  1).  This  interactive 
process  is  perfonned  in  a  collaborative  team  environment,  with  collaborative  defined  as  the 
“cognitive  aspects  of  joint  analysis  or  problem  solving  for  the  purpose  of  attaining  shared 
understanding  sufficient  to  achieve  situational  awareness  for  decision  making  or  creation  of  a 
product”  (Letsky  &  Warner,  2008,  p.  4).  Collaboration  provides  increased  information 
processing  capacity  where  more  minds  are  enlisted  to  handle  complex  problems  (Hocevar, 
Thomas,  &  Jansen,  2006).  Team  members  offer  several  perspectives  on  an  issue  for  generating, 
choosing,  and  implementing  action  plans.  A  collaborative  approach  also  ensures  greater 
flexibility  and  innovation  in  situations  where  human  judgment  and  experience  are  leveraged 
(Hocevar,  et  al,  2006.) 

In  this  paper,  we  examine  team  collaboration  in  three  distinct  task  domains  involving  two  tasks 
we  label  “execution”  tasks  and  one  planning  task:  an  air  operations  center  exercise  that  involved 
time-sensitive  targeting,  the  Federal  Aviation  Administration  (FAA)  collaborating  with  the 
Northeast  Air  Defense  Sector  (NEADS)  on  Sept  11,  2001,  and  a  planning  task  for  UAV 
utilization. 

Decision  Making 

A  decision  can  be  defined  as  a  “mental  event  that  occurs  at  a  singular  point  in  time. .  .that  leads 
immediately  or  directly  to  action ”  [italics  added]  (Hoffman  &  Yates,  2005,  p.  77).  From  this 
perspective,  a  decision  is  a  commitment  to  a  course  of  action.  Other  researchers  view  decision 
making  as  a  macrocognitive  process  that  both  supports  and  is  supported  by  deciding  (Klein, 

Ross,  Moon,  Klein,  Hoffman,  &  Hollnagel,  2003).  A  complex  problem-solving  situation 
typically  entails  many  decisions.  These  decisions  include  implementing  actions  in  response  to 
prior  conditions  or  to  the  existing  situation.  As  a  situation  unfolds,  it  will  likely  present  new 
events  requiring  a  decision. 

Dynamic  Decision-Making  Tasks 

Dynamic  decision-making  tasks,  such  as  the  decision  domains  investigated  for  this  research,  are 
characterized  by  situations  where:  (1)  A  series  of  decisions  is  needed,  that  is,  the  problem- 
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solving  event  comprises  many  decisions  to  effectively  deal  with  the  problem  as  it  unfolds.  (2) 
Decisions  are  not  independent  because  current  decisions  are  constrained  by  earlier  decisions, 
and,  in  turn  constrain  later  ones.  (3)  The  problem  state  changes  during  the  decision  process  both 
autonomously  -  as  the  situation  continues  to  unfold,  and  as  a  consequence  of  the  decision¬ 
maker’s  actions.  (4)  Decisions  are  made  in  real  time  (Brehmer,  1992).  It  is  necessary  for  the 
practitioner  to  consider  how  the  current  decision  will  solve  the  immediate  problem,  as  well  as 
how  it  will  impact  future  aspects  of  the  overall  problem-solving  task.  More  importantly,  it  is  not 
sufficient  to  make  correct  decisions  “in  the  correct  order,  they  also  need  to  be  made  at  the  correct 
moment  in  time”  (Brehmer,  1992,  p.  16).  Dynamic  decision  making  situations  are  inherently 
stressful  in  part  because  the  decision  maker  cannot  control  when  these  critical  decisions  have  to 
be  made. 

Dynamic  decision  making  tasks  are  found  across  the  spectrum  of  problem  solving  domains, 
including  process  control  plants,  patient  management  in  hospitals,  managing  a  business,  and 
fighting  a  battle.  Two  of  the  tasks  we  examined  were  dynamic  decision-making  tasks,  and  one 
was  a  planning  task.  Our  objective  was  to  compare  the  team  collaboration  that  is  entailed  in  a 
planning  task  versus  a  dynamic  decision-making  or  execution  task  to  see  if  differences  exist  in 
the  macrocognitive  processes  used  by  teams  engaged  in  two  different  types  of  tasks,  that  is, 
planning  versus  execution. 

Implementing  the  decision  often  shapes  both  the  problem  as  well  as  the  cognitive  process 
involved  in  decision  making.  An  example  from  air  warfare  would  be  where  the  tactical  action 
officer  (TAO)  recommended  issuing  a  verbal  warning  to  an  aircraft  in  the  vicinity  of  the  ship  in 
an  effort  to  obtain  additional  infonnation  on  the  possible  intent  of  the  aircraft.  However,  in  one 
particular  case,  the  more  experienced  commanding  officer  (CO)  negated  that  recommended 
action  because,  as  he  put  it,  “we  don’t  want  to  do  that  because,  he  may  not  even  see  us  here  in 
this  confined  area  and  that  may  just  serve  as  a  flag  to  the  pilot  -  there  is  a  U.S.  Navy  warship.”  In 
this  case  the  CO  was  thinking  ahead  to  how  a  decision  would  impact  the  future  situation.  This 
represents  an  example  of  anticipatory  thinking  (Klein,  Snowden,  &  Pin,  2007). 

By  implementing  a  decision,  and  obtaining  feedback  on  its  results,  the  operator  has  changed  the 
problem.  For  example,  taking  an  action  against  an  inbound  aircraft,  during  an  air  warfare 
scenario  aboard  a  US  Navy  ship,  and  observing  that  aircraft’s  response,  or  lack  of  response  to 
that  action,  will  recast  the  problem,  or  change  the  practitioner’s  mental  model  of  that  task,  in 
terms  of  detennining  the  intent  of  the  aircraft.  In  a  similar  vein,  during  a  Coast  Guard  maritime 
interdiction  operation  (MIO),  collecting  new  information  by  obtaining  the  results  of  analysis  of 
radiological  data  will  move  the  boarding  officer’s  assessment  of  the  type  of  cargo  further  along 
toward  a  resolution  of  the  problem  (that  is,  identifying  the  type  of  cargo  and  what  type  of 
response  is  required). 

A  team  can  be  defined  as  a  “small  group  of  people  with  complementary  skills  who  are 
committed  to  a  common  purpose,  performance  goals,  and  approach  for  which  they  hold 
themselves  mutually  accountable”  (Katzenbach  &  Smith,  1993,  p.  45). 

Model  of  Team  Collaboration 

In  this  paper  we  report  on  research  conducted  to  empirically  evaluate  and,  if  necessary,  refine  a 
model  of  team  collaboration  developed  by  Fiore,  Smith-Jentsch,  Salas,  Warner,  &  Letsky  (in 
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press),  depicted  in  Figure  1.  The  model  was  developed  to  describe  the  dimensions  of  macrocog¬ 
nition  where  the  emphasis  is  to  describe  these  processes  in  a  measurement  framework  and 
includes  collaborative  stages  that  the  team  goes  through  during  the  problem-solving  task,  the 
cognitive  processes  used  by  the  team,  and  final  outputs,  such  as  the  selected  course  of  action.  The 
focus  of  the  collaboration  model  is  on  developing  new  knowledge  among  the  team  members. 


Figure  1.  Model  of  Team  Collaboration  (from  Fiore,  et  al,  in  press). 

Table  1  includes  descriptions  of  the  macrocognitive  processes  included  in  this  model  of  team 
collaboration,  developed  by  Fiore,  et  al  (in  press).  Examples  of  chat  log  entries  from  an  air 
operations  center  scenario  coded  as  macrocognitive  processes  included  in  the  model  of  team 
collaboration  are  also  included  in  Table  1.  Internalized  team  knowledge  refers  to  the  collective 
current  knowledge  held  in  the  individual  minds  of  team  members.  Individual  knowledge 
building  refers  to  actions  taken  by  individuals  to  build  their  own  knowledge.  Team  knowledge 
building  is  a  process  that  includes  actions  by  teammates  to  disseminate  information  and 
transfonn  information  into  actionable  knowledge.  Externalized  team  knowledge  refers  to  the 
current  knowledge  overtly  expressed  to  other  team  members  through  communication  and/or 
artifacts  during  collaborative  team  problem  solving.  Team  problem  solving  outcomes  are 
assessments  of  the  quality  relative  to  a  team’s  problem  solving  or  plan. 

METHOD 

Definitions  of  the  macrocognitive  process  categories  developed  by  Fiore  et  al  (in  press)  were 
used  to  code  team  members’  communications.  The  first  task  domain  we  analyzed  was  the 
distributed  team  collaborative  communications  between  two  federal  agencies:  the  team  at  the 
Federal  Aviation  Administration  (FAA)  and  Northeast  Air  Defense  Sector  (NEADS),  the 
regional  headquarters  for  the  North  American  Aerospace  Defense  Command  (NORAD) 
responding  to  the  events  on  Sept.  11,  2001.  The  second  set  of  communications  data  was  the  Chat 
logs  from  an  Air  Force  research  event  in  the  Air  and  Space  Operations  Center  Dynamic  Effects 
Cell.  The  third  was  the  Chat  logs  from  an  experiment  involving  planning  for  unmanned  autono¬ 
mous  vehicle  (UAV)  utilization.  These  domains  were  selected  because  they  present  the  types  of 
complex  collaborative  problem-solving  scenarios  the  model  seeks  to  explain.  In  the  first  two 
problem-solving  tasks,  developing  and  maintaining  situation  awareness  was  particularly  difficult 
due  to  time  pressure,  high  workload,  incomplete  or  ambiguous  information,  and  the  knowledge- 
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intensive  nature  of  the  task.  Moreover,  many  military  tasks  require  several  geographically 
separated  personnel  to  quickly  form  a  cohesive  team,  where  the  team  must  be  able  to  effectively 
share  and  process  data,  information,  and  knowledge  to  facilitate  rapid  decision  making. 


Table  1.  Macrocognitive  Process  Definitions  and  Examples  from  Air  Operations  Data.1 


Stage  I:  Internalized  Team  Knowledge  Process:  Refers  to  the  collective  knowledge  held  in  the  individual 
minds  of  team  members.  Internalized  Team  Knowledge  is  measured  by  eliciting  it  from  individual  team 
members  using  methods  such  as  card  sorting,  concept  mapping,  paired  comparison  ratings,  scenario  probes. 

Team  Knowledge  Similarity:  Team  knowledge  similarity  can  involve  the  degree  to  which  differing 
roles  understand  one  another  (e.g.,  how  well  a  land/sea  vehicle  specialist  understands  a  humanitarian 
specialist),  or  how  well  the  team  members’  understand  the  critical  goals  and  locations  of  important 
resources  (shared  situation  awareness). 

-  No  coded  examples  for  AOC  data 

Team  Knowledge  Resources:  Team  members’  collective  understanding  of  responsibilities  and  resources 
associated  with  the  task. 

•  I  remember  sketchy  authentication  codes 

•  Fighter  aircraft  #2  is  out  of  position,  looks  like  other  strike  assets  are  quicker 

•  He  wouldn’t  request  to  return  to  base  (RTB),  he  tells  you  he  is  RTB 

Stage  II:  Individual  Knowledge  Building  Process:  is  a  process  which  includes  actions  taken  bv  individuals  in 
order  to  build  their  own  knowledge.  These  processes  can  take  place  inside  the  head  (e.g.,  reading,  mentally 
rotating  objects)  or  may  involve  overt  actions  (e.g.,  accessing  a  screenshot). 

Individual  Information  Gathering:  Individual  information  gathering  involves  actions  individuals 
engage  in  to  add  to  their  existing  knowledge  such  as  reading,  asking  questions,  accessing  displays,  etc. 

•  What  is  the  correct  way  to  pass  tasking  to  a  predator  to  attack? 

•  Joint  coordinating  elements  do  you  know  the  local  threat/  risk  in  the  area  and  do  you  have 
imagery  of  the  locations? 

•  Any  battle  damage  assessment  reports/imagery  post-strike  for  aircraft? 

Individual  Information  Synthesis:  Individual  information  synthesis  involves  comparing  relationships 
among  information,  context,  and  artifacts  to  develop  actionable  knowledge. 

•  Reliable  sources  report  a  known  country  bomb  component  supplier  is 

•  Awaiting  a  large  shipment  of  explosives-  It  is  suspected  that  a  certain  country  uses  this  location 
as  a  storage  facility  for  spent  fuel. 

Knowledge  Object  Development:  Knowledge  object  development  involves  creation  of  cognitive 
artifacts  that  represent  actionable  knowledge  for  the  task. 

No  coded  examples  for  AOC  data 

Stage  III:  Team  Knowledge  Building  Process:  is  a  process  which  includes  actions  taken  bv  teammates  to 
disseminate  information  and  to  transform  that  information  into  actionable  knowledge  for  team  members. 

Team  Information  Exchange:  Team  information  exchange  involves  passing  relevant  information  to  the 
appropriate  teammates  at  the  appropriate  times. 

•  Target  priority  coordinated,  entered  and  pushed  to  joint  time  sensitive  targeting  manager 

•  The  actual  snatch  and  grab  would  be  possibility  for  special  operation  force  (SOF)  but  we  would 
need  intelligence  assistance 

•  For  your  information,  this  area  is  now  under  SOF  control.  Reconnaissance  aircraft  to  provide 
over  watch,  SOF  is  now  in  contact  with  aircraft 

Team  Knowledge  Sharing:  Team  knowledge  sharing  involves  explanations  and  interpretations  shared 
between  team  members  or  with  the  team  as  a  whole. 

•  Self  defense  applies  for  hostile  acts  from  one  country  airspace  to  another 
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•  Enemy  forces  that  employ  ordnance,  electronic  attack  or  achieve  a  radar  lock  against  friendly 
forces  have  committed  a  hostile  act. 

Team  Solution  Option  Generation:  Team  solution  option  generation  describes  offering  potential 
solutions  to  a  problem. 

•  Awaiting  radiological  impact  assessment  on  watershed  if  strike  building.  Second  option  in  work 
is  destroy  local  roads  to  prevent  access  in/out. 

•  If  we  crater  the  runway  and  taxiways,  we  may  be  able  to  effectively  stop  the  target. 

•  To  shorten  timeline  for  tactical  tomahawk  we  can  launch  to  loiter.  Will  attempt  to  mitigate  with 
weaponeering 

Team  Evaluation  and  Negotiation  of  Alternatives:  Team  evaluation  and  negotiation  of  alternatives 
describes  clarifying  and  discussing  the  pros  and  cons  of  potential  solution  options. 

•  Just  throwing  this  out  there,  but  if  you  target  the  roadways,  is  there  a  chance  you  could  spook 
them  and  they  might  fire  off  their  missiles  and  run? 

Stage  IV:  Externalized  Team  Knowledge  Process:  Refers  to  facts,  relationships,  and  concents  that  have  been 
explicitly  agreed  upon,  or  not  openly  challenged  or  disagreed  upon,  by  factions  of  the  team. 

Externalized  Cue-strategy  Associations:  Externalized  cue-strategy  associations  describe  the  team’s 
collective  agreement  as  to  their  task  strategies  and  the  situational  cues  that  modify  those  strategies  (and 
how). 

•  The  dynamic  effects  cell  chief  stated  that  if  there  is  an  erect  launcher  in  a  joint  special 
operations  area  the  "rule  of  engagement”  is  to  kill  it  as  soon  as  possible  and  if  there  is  time  to 
de-conflict  with  the  teams 

•  He  mentioned  tomahawk  land  attack  missile  (TLAM)  wouldn't  be  de-conflicted  either,  but  I 
dispute  that  logic.  First,  we  wouldn't  use  a  TLAM  shot  to  kill  a  launcher 

•  I  don't  think.  Unless  it  was  a  last  resort. 

•  Can  get  special  operation  force  team  to  location  as  additional  resource  if  we  elect  to  monitor 
the  site  for  any  potential  leadership  meetings  that  may  occur  later 

Pattern  Recognition  and  Trend  Analysis:  Pattern  recognition  and  trend  analysis  is  the  accuracy  of  the 
patterns  or  trends  explicitly  noted  by  members  of  a  team  that  is  either  agreed  upon  or  unchallenged  by 
other  team  members. 

•  Looks  like  this  target  may  be  similar  to  our  first  target  with  regards  to  unknown  presence  of 
radiological  containers  in  facility.  We  would  look  at  interdiction  for  containment  to  prevent 
travel  to/fm  that  site,  your  thoughts  on  best  plan/option. 

Uncertainty  Resolution:  Uncertainty  resolution  is  the  degree  to  which  a  team  has  collectively  agreed 
upon  the  status  of  problem  variables  (e.g.,  hostile/friendly). 

•  Tomahawk  land  attack  missile  most  definitely  have  to  be  de-conflicted  even  for  over  flight  of  the 
joint  special  operations  area  unless  direct  otherwise  by  the  Joint  Force  Commander. 

Stage  V:  Team  Problem  Solving  Outcomes:  Are  assessments  of  duality  relating  to  a  team’s  problem  solutions 
or  plan. 

Quality  of  Plan:  Quality  of  plan  (problem  solving  solution)  involves  the  degree  to  which  the  solution 
adopted  by  a  problem  solving  team  achieves  a  resolution  to  the  problem  (e.g.,  limit  fatalities,  limit 
destruction). 

-  No  coded  examples  for  AOC  data 

Efficiency  of  Planning  Process:  Efficiency  of  planning  process  describes  the  amount  of  time  it  takes  a 
problem  solving  team  to  arrive  at  a  successful  resolution  to  a  problem. 

-  No  coded  examples  for  AOC  data 

Efficiency  of  Plan  Execution:  Efficiency  of  plan  execution  describes  the  quality  of  the  plan  (e.g., 
number  of  lives  saved)  divided  by  the  amount  of  resources  used  to  accomplish  this  and  the  amount  of 
time  the  plan  takes  to  unfold. 

-  No  coded  examples  for  AOC  data 

(Definitions  of  macrocognitive  processes  from  Fiore  et  al.,  in  press.) 
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Task  Domain  I:  Federal  Aviation  Administration  and  Northeast  Air  Defense  Sector 


On  Sept  11,  2001,  defense  of  the  U.S.  airspace  depended  on  close  interaction  between  two 
federal  agencies:  the  team  at  the  Federal  Aviation  Administration  (FAA)  and  Northeast  Air 
Defense  Sector  (NEADS),  the  regional  headquarters  for  the  North  American  Aerospace  Defense 
Command  (NORAD).  NEADS  was  the  key  command  and  control  (C2)  center  for  the  U.S. 
military  response  during  the  terrorist  attacks. 

Task.  The  airspace  of  the  continental  US  is  monitored  and  protected  by  two  main  entities:  the 
FAA,  who  is  charged  with  navigating  and  controlling  air  traffic,  and  NORAD,  who  is  charged 
with  protecting  the  North  American  continent  against  air  attack.  In  discharging  its 
responsibilities,  NORAD  utilizes  the  airspace  that  is  monitored  by  the  FAA  and  must  conduct 
their  operations  according  to  FAA  regulations  and  with  their  active  participation  (Memorandum 
of  understanding  between  FAA  and  NORAD,  1987).  Thus,  close  coordination  between  the  FAA 
and  NORAD  is  required  in  order  to  maintain  safety  of  the  U.S.  airspace. 

Task  Domain  II:  Air  and  Space  Operations  Center 

A  high-fidelity  research  event  was  conducted  with  Air  Force  personnel  in  the  Air  and  Space 
Operations  Center  Dynamic  Effects  Cell.  Dynamic  targeting  is  a  means  by  which  coalition  forces 
could  possibly  respond  to  the  employment  of  improvised  explosive  devices  (IEDs)  and  insurgent 
network  leadership  located  within  Afghanistan  and  Iraq.  Due  to  their  short  time-sensitive  nature, 
dynamic  targets  are  normally  vetted  quickly  and  accurately  through  the  targeting  cycle.  This 
process  allows  the  Joint  Force  Air  Component  Commander  (JFACC)  to  task  available  assets  or 
reassign  assets  to  engage  and  destroy  known  targets  or  other  potential  threats  at  a  moment’s 
notice.  The  limited  time  window  available  for  target  engagement  and  destruction  means  that 
coalition  forces  must  be  prepared  to  apply  timely  and  accurate  measures  and  counter-measures 
against  the  enemy. 

Participants.  Thirty-six  key  operational  command  and  control  military  and  civilian  personnel 
from  all  services  participated  in  the  dynamic  targeting  exercise.  Participants  included  personnel 
from  the  Air  Force  Warfare  Center,  Naval  Strike  Air  Warfare  Center,  Special  Operations 
Command,  and  United  States  Army  (Air  Force  Research  Laboratory  Warfighter  Readiness 
Division,  2008).  The  purpose  was  to  assess  tactics,  techniques  and  procedures  of  operational 
command  personnel  performing  kinetic  and  non-kinetic  dynamic  targeting  in  a  highly 
asymmetric  environment.  The  research  event  was  a  simulation  of  a  12-hour  overnight  shift  in  the 
dynamic  effect  cells  of  a  typical  air  operations  center  (AOC).  Communication  between  the 
various  players  was  facilitated  by  the  use  of  the  Joint  Automated  Deep  Operations  Coordination 
System  (JADOCS).  JADOCS  provides  the  warfighter  with  a  timely,  accurate,  detailed 
battlespace  view  for  planning,  coordination,  and  execution  of  targets.  It  is  a  joint  mission 
management  software  application  that  provides  a  suite  of  tools  and  interfaces  for  horizontal  and 
vertical  integration  across  battlespace  functional  areas  (Raytheon  Company,  2008). 
Communications  were  recorded  across  fourteen  different  chat  room  channels. 

Task.  Each  of  the  operational  players  had  varied  access  and  responsibilities  in  the  chat  rooms 
used  for  the  exercise.  All  players  did  not  have  access  to  every  individual  chat  room;  some  key 
operational  players  were  designated  as  room  owner,  active  participant  and/or  observer.  The 
allocation  of  assets  to  destroy  key  enemy  leadership,  suppression  of  enemy  air  defense  systems, 
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convoy  protection,  close  air  support,  and  dynamic  targeting  are  just  a  few  of  many  ways  by 
which  the  JFACC  could  use  the  assigned  assets  to  render  an  adversary’s  method  of  attack 
ineffective  and  thereby  minimize  damage  and  coalition  force  and  civilian  casualties. 

UAV  Planning  Task.  Chat  logs  from  a  laboratory  experiment  were  coded  to  provide  a  set  of 
comparison  data  for  a  planning  task.  Three  college  students  served  as  participants  and  were 
trained  to  engage  in  an  unmanned  aerial  vehicle  (UAV)  simulation  task  where  the  task  was  to 
control  their  UAVs  to  obtain  photos  of  assigned  sites. 

Coding  of  Team  Communications 

Communications  between  the  FAA  and  NEADS  teams  were  captured  as  audio  recordings  and 
subsequently  transcribed.  The  audio  from  one  of  the  primary  communications  channels  used  by 
the  mission  control  commander  (MCC),  channel  three  -  MCC  Operations,  was  professionally 
transcribed.  The  resulting  transcript  provides  a  realistic  example  of  an  inter-agency  collaborative 
team’s  response  to  a  real-world  emergency.  Chat  logs  were  used  from  the  AOC  exercise.  Two 
graduate  students  coded  the  FAA/NEADS  data  and  another  pair  of  graduate  students  coded  the 
AOC  data  to  establish  the  reliability  of  our  coding  method.  Subsequently,  one  of  the  authors 
coded  the  team  communications  in  the  chat  log  from  the  UAV  planning  experiment. 

To  determine  the  overall  percentage  of  agreement  between  the  two  coders,  the  qualitative 
categorical  statistic  Cohen’s  Kappa  was  used.  An  additional  code  was  created  called  the  Extra 
Code  Filler  (ECF)  which  was  used  to  indicate  where  one  coder  did  not  specify  a  separate  code 
for  an  utterance.  This  was  necessary  to  ensure  that  each  coder  assigned  the  exact  same  number  of 
total  codes  in  order  to  accurately  calculate  inter-rater  reliability  for  the  two  coders,  using 
Cohen’s  Kappa  coefficient.  Cohen’s  Kappa  is  the  preferred  statistic  over  the  Chi-square  statistic 
as  kappa  tests  for  agreement  whereas  Chi-square  tests  for  association  (Thomas,  &  Hersen,  2003). 
Cohen’s  Kappa  accounts  for  and  factors  into  the  calculation  that  each  coder  may  also  agree  by 
chance  and  not  strictly  because  they  chose  the  same  selection  option  or  code. 

Communications  were  segmented  into  utterances;  that  is,  elliptic  or  sentential  constructions  that 
referred  to  a  distinct  macrocognitive  process.  Each  utterance  was  typically  given  a  separate 
code;  however,  in  some  instances  two  utterances  referred  to  the  same  cognitive  process  and 
were  thus  grouped  under  one  code.  One  week  prior  to  coding  the  FAA/NEADS  and  the  AOC 
communications,  two  raters  coded  team  communications  data  from  a  maritime  interdiction  operation 
exercise  not  included  in  the  present  data  set.  Raters  discussed  their  respective  coding  with  one  of  the 
authors  to  calibrate  their  use  of  the  macrocognitive  process  categories.  Following  this  training  period 
they  independently  coded  the  FAA/NEADS  transcript  and  the  AOC  Chat  logs,  subsequently  reviewed 
their  coding,  calculated  percent  agreement,  and  resolved  any  differences  in  coding. 

Inter-Rater  Reliability.  A  pair  of  coders  coded  2,493  and  2,278  utterances  for  the  AOC  and 
FAA/  NEADS  data,  respectively.  Inter-rater  agreement  was  89.32%  and  77%,  for  the  AOC  and 
FAA/  NEADS  data,  respectively.  One  coder  coded  the  Chat  log  data  for  the  UAV  planning 
experiment. 


RESULTS 

Consistent  with  naturalistic  decision  making  research  (e.g.,  Fischer,  Orasanu,  &  Montalvo,  1993; 
Klein,  1989;  1993;  Mosier  1991;  Orasanu  &  Connolly,  1993;  Rasmussen,  1993)  we  found  that 
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problem  solving  by  the  FAA  and  NEADS  on  Sept  11,  2001,  the  air  operations  center  dynamic 
targeting  scenario,  and  the  UAV  planning  tasks  comprised  a  series  of  decisions.  The  most 
significant  finding  is  that  a  new  macrocognitive  process  emerged  during  the  coding  process  to 
account  for  team  communications/  chat  log  entries  where  a  team  member  made  a  Decision  to 
Take  Action  (DTA):  A  team  member  either  issued  an  order  for  a  course  of  action  (COA)  or 
make  a  request  for  a  team  member  to  do  something  (that  is,  request  take  action).  These  two  types 
of  decisions  are  differentiated  by  the  authority  relationship  between  the  speaker  and  receiver  and 
the  criticality  of  the  action  to  be  taken.  A  course  of  action  is  issued  by  a  superior  to  a  subordinate 
and  the  action  to  be  implemented  is  more  critical  to  the  outcome  of  the  scenario.  Requesting  a 
peer  to  take  action  involves  a  lower  level  type  of  decision  in  terms  of  the  authority  relationship, 
that  is,  peer-to-peer,  and  the  less  critical  nature  of  the  action  to  the  outcome  of  the  scenario. 

Initially  we  did  not  anticipate  team  members  involved  in  the  UAV  planning  task  to  make  the 
same  types  of  decisions  that  required  immediate  action.  However,  although  this  was  labeled  a 
planning  task  it  actually  involved  dynamic  planning  and  re-planning  while  the  participants  were 
performing  their  tasks  involved  in  navigating  their  UAV  to  obtain  photos  of  designated  sites. 
Decisions  were  made  iteratively  throughout  the  entire  problem  situation  for  the  two  tasks  that  we 
label  “execution,”  or  dynamic  decision  making  tasks  as  well  as,  to  a  lesser  degree,  for  the 
dynamic  UAV  planning  task.  This  UAV  task  also  entailed  execution  of  the  task.  The 
participants  were  engaged  in  “on-the-fly”  planning  based  on  directions  from  experimenters 
regarding  which  target  to  photograph  next.  This  dynamic  planning  and  execution  task  is  distinct 
from  a  strictly  planning  task  that  culminates  in  team  members  agreeing  on  one  big  final  decision 
(i.e.,  the  plan)  that  is  typical  of  a  planning  task.  Based  on  these  findings,  we  recommend  the 
Fiore,  Smith- Jentsch,  Salas,  Warner,  and  Letsky  (in  press)  model  of  team  collaboration  be 
revised  to  include  a  new  category:  Decision  to  take  action. 

Table  2  presents  the  results  of  the  coding  of  three  tasks:  two  execution  tasks  and  one  planning 
task.  Coders  used  additional  categories  to  code  the  team  communications  for  all  three  task 
domains;  these  include  administrative,  miscellaneous  and  extra  code  filler  (ECF).  While  speech 
turns  coded  as  administrative  and  miscellaneous  contribute  to  effective,  closed-loop  commun¬ 
ications  they  do  not  represent  a  cognitive  process  and  were  thus  removed  for  this  analysis.  Extra 
code  filler  was  a  coding  that  was  added  to  utterances  where  one  coder  had  not  coded  an  utterance 
and  the  other  coded  had  to  ensure  an  equal  number  of  codes  between  the  two  coders. 

Table  2  includes  the  percentages  of  the  macrocognitive  processes  used  when  the  administrative, 
miscellaneous,  and  ECF  codes  were  removed  from  the  calculations.  Team  Information 
Exchange,  51.5,  52.9,  and  58  percent,  respectively  for  the  AOC,  NORAD,  and  UAV  data,  and 
Individual  Information  Gathering,  21.9,  30.9,  and  32.6  percent,  for  the  AOC,  NORAD,  and  UAV 
data,  had  the  highest  usage.  In  general,  the  large  number  of  utterances/  chat  log  entries  coded  as 
Team  Information  Exchange  and  Individual  Information  Gathering  indicate  the  huge  emphasis 
on  knowledge  construction  that  is  required  at  both  the  individual  and  team  level  for  all  three 
tasks.  Decision  to  Take  Action  (a  combination  of  Course  of  Action  and  Request  Take  Action 
codes)  encompasses  10.3,  5.5,  and  4.3  percent,  respectively  for  the  AOC,  NORAD,  and  UAV 
data.  Team  Knowledge  Sharing,  the  fourth  highest  used  macrocognitive  process  was  used  10.3, 
3.7,  and  4.3  percent,  respectively. 

Individual  Knowledge  Building  involves  actions  taken  at  the  intra-individual  level  for  the 
purpose  of  building  one’s  own  knowledge.  This  may  involve  reading  task  relevant  content, 
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therefore,  taking  place  in  the  head  of  the  individual,  or  it  may  involve  interactions  with  a  system 
such  as  accessing  a  screenshot.  These  actions  are  hypothesized  to  involve  Individual  Information 
Gathering,  Individual  Information  Synthesis,  and  Knowledge  Object  Development.  These 
processes  involve  a  number  of  activities  ranging  from  reading,  to  question  asking,  to  accessing 
displays.  It  is  noteworthy  that  the  vast  majority  of  the  communications  coded  as  Individual 
Information  Gathering  involved  instances  of  a  team  member  asking  a  question.  We  interpret  this 
as  an  indication  of  the  high  degree  of  ambiguity  inherent  in  the  three  task  domains  reported  on  in 
this  paper.  For  example,  the  AOC  exercise  placed  team  members  in  a  time-compressed  situation 
that  required  the  operators  to  process  a  large  amount  of  information  from  various  intelligence 
sensors.  Team  members  engaged  in  building  their  individual  knowledge  by  asking  lots  of 
questions  to  continue  to  build  on  their  existing  knowledge  and  to  maintain  situation  awareness  in 
this  dynamic  environment.  Similar  results,  where  teams  rely  heavily  on  Individual  Information 
Gathering  and  Team  Information  Exchange  have  been  reported  for  several  tasks  including  the 
Fire  Department  of  New  York  responding  o  the  events  of  Sept  11,  2001  (Hutchins  &  Kendall, 
2008)  and  air  warfare  decision  making  in  the  combat  information  center  of  Navy  ships  (Hutchins 
&  Kendall,  in  press). 


Table  2.  Percentage  of  Macrocognitive  Process  Codes  for  Three  Tasks. 


Code 

Macrocognitive  Process  Categories 

Percentage  of  Speech  Turns 

Individual  Knowledge  Building 

AOC 

NORAD 

UAV  Plan’g 

IIG 

Individual  Information  Gathering 

21.93 

30.91 

32.57 

IIS 

Individual  Information  Synthesis 

3.63 

1.74 

0.75 

KOB 

Knowledge  Object  Development 

0.00 

0.00 

0.00 

Team  Knowledge  Building 

TIE 

Team  Information  Exchange 

51.45 

52.87 

58.00 

TKS 

Team  Knowledge  Sharing 

10.33 

3.74 

4.29 

TSOG 

Team  Solution  Option  Generation 

0.90 

3.07 

0.00 

TENA 

Team  Evaluation  and  Negotiation  of  Alternatives 

0.49 

0.00 

0.00 

TPPR 

Team  Process  and  Plan  Regulation 

0.00 

0.00 

0.00 

Internalized  Team  Knowledge 

ITK 

Team  Knowledge  Similarity 

0.08 

0.00 

0.00 

TKR 

Team  Knowledge  Resources 

0.16 

0.00 

0.00 

IK 

Inter-positional  Knowledge  (3) 

0.06 

0.20 

0.00 

ISA 

Individual  Situational  Awareness  (1) 

0.00 

1.67 

0.00 

Externalized  Team  Knowledge 

ECS  A 

Externalized  Cue-Strategy  Association 

0.49 

0.07 

0.00 

PRTA 

Pattern  Recognition  and  Trend  Analysis 

0.16 

0.07 

0.00 

UR 

Uncertainty  Resolution 

0.08 

0.13 

0.00 

Problem  Solving  Outcomes 

QOP 

Quality  of  Plan 

0.00 

0.00 

0.00 

EPP 

Efficiency  of  Planning  Process 

0.00 

0.00 

0.00 

EPE 

Efficiency  of  Planning  Execution 

0.00 

0.00 

0.00 

Decision  to  Take  Action 

DTA 

DTA  (Issue  Course  of  Action) 

6.61 

4.27 

0.00 

RTA 

DTA  (Request  Take  Action) 

3.69 

1.27 

4.29 
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Inter-Rater  Reliability 


The  high  Kappa  Cohen  coefficient  achieved  by  both  pairs  of  coders  indicates  the  two  coder’s 
agreement  is  substantial  (Landis  and  Koch,  1977).  The  high  inter-rater  reliability  kappa  score 
indicates  the  macrocognitive  process  definitions  used  by  the  coders  are  objective.  There  is  also 
evidence  that  most  of  the  macrocognitive  processes  in  this  model  of  team  collaboration  are 
applicable  to  real  world  decision  making  domains. 

Table  3  presents  the  results  of  the  Kappa  analysis,  for  the  AOC  data,  in  the  form  of  a  pivot  table 
which  compares  coder  1  codes  against  coder  2  codes.  Coder  1  codes  are  displayed  in  the 
columns  and  coder  2  codes  are  read  across  rows.  Coder  matches  run  diagonally  through  the 
pivot  table,  in  the  bolded  cells,  starting  with  Administrative  183,  Miscellaneous  663,  Team 
Information  Exchange  1 134,  and  so  on.  The  pivot  table  also  highlights  which  codes  the  coders 
disagreed  upon.  For  example,  under  the  individual  information  gathering  (IIG)  category,  coder  1 
and  coder  2  had  assigned  a  total  of  537  and  526  IIG  codes,  respectively,  to  the  data.  However, 
both  coders  matched  selections  for  521  of  the  individual  information  gathering  (IIG)  codes. 
There  were  16  and  five  instances,  respectively,  where  coder  1  and  coder  2  disagreed  with  the 
other  coder.  Reading  down  the  column  under  IIG  one  sees  that  coder  1  had  applied  five 
individual  information  gathering  (IIG)  codes,  one  each  for  team  knowledge  sharing  (TKS), 
Course  of  Action  (COA),  request  take  action  (RTA)  and  applied  eight  ECF  codes.  Similarly  for 
coder  2,  of  the  five  codes  that  differed  between  the  two  coders  for  IIG,  two  each  were  coded  as 
ADMIN,  and  team  information  exchange  (TIE),  and  one  team  knowledge  sharing  (TKS). 

In  Table  3,  the  diagonal  cells  of  the  matrix  indicate  agreement  between  the  coders  whereas  the 
values  in  the  other  cells  indicate  the  difference  between  what  each  of  the  coders  chose.  As  an 
example,  in  the  pivot  table,  both  raters  agreed  1134  times  on  TIE,  where  coder  1  applied  this 
code  1,192  times,  to  58  additional  utterances  and  coder  2  applied  it  1187  times,  or  to  53 
additional  utterances. 

When  there  was  a  disagreement  on  Team  Information  Exchange  (TIE)  or  Team  Knowledge 
Sharing  (TKS)  codes,  the  other  code  typically  selected  code  was  Team  Knowledge  Sharing  and 
Team  Information  Exchange  (respectively).  Reading  down  the  pivot  table  for  category  TIE,  one 
sees  the  number  of  agreed  upon  Team  Information  Exchange  codes  is  1,134  and  that  coder  2 
disagreed  with  coder  1  and  selected  Team  Knowledge  Similarity  17  times.  Additionally,  reading 
down  Team  Knowledge  Sharing  (TKS)  category,  both  coders  agreed  136  times  but  coder  2 
disagreed  and  selected  Team  Information  Exchange  24  times.  The  disagreement  between  TKS 
and  TIE  and  the  patterned  alternative  response  of  the  other  coder  code  indicates  that  there  is 
some  ambiguity  in  the  measurement  model  definition  for  both  of  these  macrocognitive 
processes.  Furthermore,  when  coder  1  selected  Team  Information  Exchange  and  Team 
Knowledge  Sharing,  coder  2  disagreed  and  selected  the  Extra  Code  Filler  (ECF)  code  20  and  39 
times  (respectively).  This  disagreement  between  coders  and  non  selection  of  macrocognitive 
process  definition  indicates  that  the  definitions  for  Team  Knowledge  Sharing  and  Team 
Information  Exchange  need  to  be  refined  to  clarify  remove  any  ambiguity. 

Cases  where  there  are  large  disagreements  between  two  coders  indicate  the  definitions  for  the 
macrocognitive  process  are  not  mutually  exclusive  and  these  definitions  need  to  be  refined.  For 
example,  the  two  coders  agreed  on  136  codes  for  team  knowledge  sharing  (TKS)  where  coder  1 
applied  TKS  a  total  of  209  times  and  coder  2  applied  TKS  172  times.  Looking  across  the  row 
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labeled  TKS  one  can  see  the  other  codes  that  were  applied  to  utterances  by  coder  2  when  coder  1 
applied  TKS.  There  were  seventeen  instances  where  coder  2  used  team  information  exchange 
(TIE)  instead  of  team  knowledge  sharing  (TKS). 


Table  3.  Coder  Pivot  Table  Showing  Kappa  Results  for  the  Air  Operations  Center  Data. 


CODE 

TITLE 

ADMIN 

MISC 

TIE 

IIG 

t fl 

HH 

HH 

ECSA 

TKS 

COA 

RTA 

TSOG 

TENA 

ITK 

UR 

PRTA 

ECF 

Total  Coder  2 

ADMIN 

183 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

1 

185 

MISC 

0 

663 

6 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

670 

TIE 

0 

1 

1134 

5 

9 

1 

24 

9 

0 

2 

0 

0 

0 

1 

1 

1187 

IIG 

2 

0 

2 

521 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

526 

IIS 

0 

0 

4 

0 

23 

0 

6 

0 

0 

0 

0 

0 

0 

0 

0 

33 

ECSA 

0 

0 

0 

0 

0 

4 

0 

0 

0 

0 

0 

0 

0 

0 

0 

4 

TKS 

0 

0 

17 

1 

6 

1 

136 

3 

0 

3 

3 

1 

0 

1 

0 

172 

COA 

0 

0 

4 

1 

0 

0 

1 

138 

1 

1 

0 

0 

0 

0 

3 

149 

RTA 

0 

0 

4 

1 

0 

0 

1 

0 

78 

0 

0 

0 

0 

0 

3 

87 

TSOG 

0 

0 

1 

0 

1 

2 

0 

0 

0 

6 

1 

0 

0 

0 

0 

11 

TENA 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

3 

0 

0 

0 

0 

4 

ITK 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

3 

0 

0 

0 

3 

UR 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

PRTA 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

ECF 

0 

2 

20 

8 

33 

4 

39 

3 

2 

6 

5 

2 

1 

2 

0 

127 

Otal  Code 
1 

185 

666 

1192 

537 

72 

12 

209 

154 

81 

19 

12 

6 

1 

4 

8 

3158 

Table  4  presents  an  excerpt  of  individual  knowledge  building  from  the  Air  Operations  Center 
dynamic  targeting  scenario  that  includes  several  instances  of  individual  information  gathering, 
individual  information  synthesis,  and  team  information  exchange. 


Table  4.  Individual  Knowledge  Building:  Examples  from  Air  Operations  Center. 


Individual  Knowledge  Building 

Originator 

Communication 

Code 

C2DO 

SIDO:  I  missed  the  name  of  the  airfield? 

IIG 

C2DO 

SIDO:  Type  of  aircraft  we  are  looking  for  and  its  latitude  and  longitude? 

IIG 

SODO 

Command  control  duty  officer  (C2DO),  what  is  the  capability  to  track  STOL 

Cargo  aircraft  with  (Tac  C2)? 

IIG 

SIDO 

C2DO,  STOL  cargo  aircraft  has  departed  to  the  target;  may  operate  between  44 
and  200  knots. 

IIS 

IOT 

NTI:  What  can  you  tell  me  about  bomb  supplier  #1? 

IIG 

NTI 

Information  operations  targeteer,  bomb  supplier  #1  is  a  known  materials  supplier; 

IIS/ 

communication  on  cover  and  will  report  any  new  intelligence  when  available. 

TIE 

C2C0:  Command  and  Control  Officer  SODO:  Special  Operations  Duty  Officer 
SIDO:  Senior  Intelligence  Duty  Officer  IOT:  Information  Operations  Targeteer 
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Team  Knowledge  Building 

Team  Knowledge  Building  includes  actions  taken  by  team  members  to  disseminate  and  transform 
information  into  actionable  knowledge.  To  be  effective  in  the  highly  dynamic  environment  of 
time-sensitive  targeting,  AOC  team  members  must  be  ready  to  make  rapid  decisions  for  difficult 
problems  in  a  time-compressed  environment.  The  data  indicate  team  members  engaged  in  a  large 
percentage  of  information  sharing  to  keep  all  team  members  aware  of  the  evolving  situation. 
Table  5  provides  examples  of  team  information  exchange  where  team  members  discussed  the 
effects  of  radiological  fallout  from  a  possible  strike  against  a  building.  They  also  provided  an 
alternate  solution  option  -  team  solution  option  generation  (TSOG).  Team  Knowledge  Building 
is  essential  to  develop  and  maintain  team  situation  awareness  that  provides  the  foundation  for 
informed  decision  making. 

Table  5.  Team  Knowledge  Building:  Team  Information  Exchange  Example. 


Team  Knowledge  Building: 

Team  Information  Exchange  and  Solution  Option  Generation 

Originator 

Communication 

Code 

DEC 

Awaiting  radiological  impact  assessment  on  watershed  if  the  building  is  to  be  strike. 
Second  option  in  work  is  to  destroy  local  roads  to  prevent  access  in/out. 

TIE 

TSOG 

DECD 

Aircraft  returns  watershed  non-issue 

TIE 

SIDO 

Airfield  is  located  at  (*Removed*);  type  of  aircraft  is  STOL  cargo  plane. 

TIE/TIE 

JOCJCE 

Dynamic  effects  cell,  you  have  high-value  target  on  your  dynamic  target  list.  What 
is  the  air  combat  commander  game  plan?  If  you  have  a  good  one,  I  will  appoint  you 
the  lead  but  1  think  SOF  needs  to  be  considered. 

TIE 

IIG 

DEC:  Dynamic  Effects  Cell  DEDC:  Dynamic  Effects  Cell  Deputy  JOC  JCEE:  Joint  Operations  Center 


A  total  of  63.17,  59.58,  and  62.29  percent,  respectively,  of  the  team  communications  for  the  AOC, 
NORAD,  and  UAV  tasks,  were  coded  as  one  of  the  macrocognitive  processes  that  occur  during  the 
Team  Knowledge  Building  Stage.  Macrocognitive  processes  that  were  used  by  the  team  include 
Team  Information  Exchange  (TIE),  Team  Knowledge  Sharing  (TKS),  Team  Solution  Option 
Generation  (TSOG),  and  Team  Evaluation  and  Negotiation  of  Alternatives  (TENA).  The  first  entry 
in  Table  5  provides  an  example  of  team  information  exchange,  “Awaiting  radiological  impact 
assessment  on  watershed  if  the  building  is  to  be  strike,”  followed  by  an  example  of  Team  Solution 
Option  Generation,  “Second  option  in  work  is  to  destroy  local  roads  to  prevent  access  in/out.  Table 
6  presents  an  excerpt  from  the  AOC  chat  log  data  that  illustrates  team  members  sharing  information 
on  rules  of  engagement  and  discussing  the  effects  of  a  strike  mission  against  an  airfield. 


Table  6.  Macrocognitive  Processes  during  Team  Knowledge  Building  Stage  of  Collaboration. 


Team  Knowledge  Building 

Originator 

Communication 

Code 

DEC 

Self  defense  applies  for  hostile  acts  from  Country  #3  fighters  in  Country  #  2  or  #4  airspace. 

TKS 

DEC 

Enemy  forces  that  employ  ordnance,  electronic  attack  or  achieve  a  radar  lock 
against  friendly  forces  have  committed  a  hostile  act 

TKS 

TDO 

If  we  crater  the  runway  and  taxiways,  we  may  be  able  to  effectively  stop  the  target. 

TSOG 

IOT 

Target  Duty  Officer  (TDO):  Just  throwing  this  out  there,  but  if  you  target  the  roadways,  is 
there  a  chance  you  could  spook  them  and  they  might  fire  off  their  missiles  and  run? 

TENA 

DEC:  Dynamic  Effects  Cell  TDO:  Target  Duty  Officer  IOT:  Information  Operations  Targeteer 


15 


Externalized  Team  Knowledge 

Externalized  Team  Knowledge  refers  to  knowledge  that  has  been  agreed  upon  by  members  of  the 
team.  Table  7  presents  examples  of  the  macrocognitive  processes  employed  during  the 
Externalized  Team  Knowledge  stage  from  the  AOC  data.  Externalized  Cue-Strategy  Associations 
(ECS A)  describe  the  team’s  collective  agreement  as  to  their  task  strategies  and  the  situational 
cues  that  modify  those  strategies  (and  how).  An  example  of  this  is  found  in  the  first  entry  of 
Table  7,  where  the  operator  says,  “The  dynamic  effects  cell  chief  stated  that  if  there  is  an  erect 
launcher  in  a  joint  special  operations  area  (JSOA)  his  rules  of  engagement  (ROE)  are  to  kill  it  as 
soon  as  possible  and  if  there  is  time,  to  de-conflict  with  the  teams.”  An  example  of  Uncertainty 
Resolution  (UR)  is  found  in  #6,  Table  7,  where  the  Joint  Special  Operations  Force  Targeteer 
says:  “TLAMs  most  definitely  have  to  be  de-conflicted  even  for  over  flight  of  the  JSOA,  unless 
directed  otherwise  by  the  JFC.” 

Pattern  Recognition  and  Trend  Analysis  (PRTA)  refers  to  the  accuracy  of  the  patterns  or  trends 
explicitly  noted  by  members  of  a  team  that  is  either  agreed  upon  or  unchallenged  by  other  team 
members.  A  member  of  the  dynamic  effects  cell  provides  an  example  of  PRTA,  entry  #7,  with 
the  following:  “Looks  like  this  target  may  be  similar  to  our  first  target  with  regards  to  unknown 
presence  of  radiological  containers  in  facility.  We  would  look  at  interdiction  for  containment  to 
prevent  travel  to/from  that  site,  your  thoughts  on  best  plan/option.” 


Table  7.  Examples  of  Externalized  Team  Knowledge  from  Air  Operations  Center  Data. 


Externalized  Team  Knowledge 

No. 

Originator 

Communication 

Code 

1. 

DECSOLE 

The  dynamic  effects  cell  chief  stated  that  if  there  is  an  erect  launcher  in  a  joint 
Special  operations  area  (JSOA)  his  rules  of  engagement  (ROE)  are  to  kill  it  as  soon 
as  possible  and  if  there  is  time,  to  de -conflict  with  the  teams. 

ECS  A 

2. 

JSOFT 

Correct,  if  per  joint  force  commander  (JFC) 

TIE 

3. 

DECSOLE 

I  can't  remember  ROE  in  the  west  for  OIF  but  1  think  it  was  something  similar. 

TIE 

4. 

DECSOLE 

He  mentioned  tomahawk  land  attack  missiles  (TLAM)  wouldn't  be  de-conflicted 
either,  but  I  dispute  that  logic.  First,  we  wouldn't  use  a  TLAM  to  shot  a  launcher. . . 

I  don't  think.  Unless  it  was  a  last  resort. 

ECS  A 

5. 

SECSOLE 

Second  the  flight  time  is  great  enough  to  pass  the  warning  and  do  the  de- 
confliction. 

TIE 

6. 

JSOFT 

TLAMs  most  definitely  have  to  be  de-conflicted  even  for  over  flight  of  the  JSOA, 
unless  directed  otherwise  by  the  JFC.  He's  not  the  JFC.  If  any  issues,  let  me  know 

UR/ 

and  I'll  pass  up  to  the  joint  special  operation  task  force  commander  for  discussion 
with  the  JFC. 

ECS  A 

7.  DEC 

Looks  like  this  target  may  be  similar  to  our  first  target  with  regards  to  unknown 
presence  of  Radiological  containers  in  facility.  We  would  look  at  interdiction  for 
containment  to  prevent  travel  to/fm  that  site,  your  thoughts  on  best  plan/option 

PRTA 

DECSOLE:  Dynamic  Effects  Cell  Special  Operations  Liaison  Officer 
JSOFT:  Joint  Special  Operations  Force  Targeteer 

SECSOLE:  Special  Operations  Liaison  Officer  DEC:  Dynamic  Effects  Cell 


Table  8  presents  an  example  of  how  information  was  shared  as  a  series  of  team  information 
exchanges  (TIE)  and  individual  information  gathering  (IIG).  That  is,  requests  for  information 
were  integrated  and  lead  Huntress  to  decide  on  a  major  course  of  action,  without  solving  the 
overall  problem.  Specifically,  Team  Information  Exchange  led  to  Individual  Information 
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Gathering,  which  led  to  Decision  to  Take  Action  (Course  of  Action),  “Intercept.  Intercept.  We 
want  the  aircraft  away  from  there,”  and  then  the  problem  continues. 


Table  8.  Excerpt  from  NEADS  data  to  illustrate  the  Dynamic  Decision-Making  Process. 


Line 

Speaker 

Message 

Code 

702 

Male  Speaker. 

Sergeant  Demage? 

MISC 

We  are  working  a  tanker. 

TIE 

703 

Sergeant  M: 

There  is  a  bomb  on  board  Boston  - 

TIE 

704 

Male  Speaker. 

*Expletive* 

MISC 

705 

Male  Speaker: 

On  board  what?  Boston  93? 

IIG 

706 

Huntress'. 

United. 

TIE 

707 

Male  Speaker. 

United? 

IIG 

708 

Sergeant M: 

Bomb  on  board  United  93. 

TIE 

709 

Huntress'. 

Intercept.  Intercept.  We  want  the  aircraft  away  from  there. 

DTA  (COA) 

DISCUSSION 

A  new  macrocognitive  process  emerged  during  coding.  Decision  to  Take  Action  (DTA)  is 
considered  to  be  both  a  macrocognitive  process  and  a  product  (Klein,  1993).  This  finding 
indicates  a  non-exhaustive  set  of  macrocognitive  processes  in  the  model  of  team  collaboration. 
These  results  corroborate  results  from  other  analyses  based  on  analysis  and  coding  transcripts 
from  other  dynamic  decision-making  tasks  (Hutchins  &  Kendall,  in  press;  Hutchins  &  Kendall, 
2008).  For  example,  in  a  30-minute  air  warfare  scenario,  nine  decisions  to  take  action  were  made 
by  the  air  warfare  team.  Moreover,  nineteen  percent  of  the  team  communications  from  the  Fire 
Department  of  New  York  responding  to  the  Sept  11,  2001,  attacks  on  the  World  Trade  Center 
involved  decisions  to  take  action. 

Coding  schemes  should  be  mutually  exclusive,  exhaustive,  and  equivalent.  The  high  Cohen 
Kappa  coefficient  indicates  the  two  coder’s  agreement  is  substantial  and  indicates  the  macrocog¬ 
nitive  process  definitions  are  objective.  The  pivot  table  of  results  highlights  instances  where  the 
two  coders  disagreed  and  shows  which  codes  were  used  by  the  other  coder  when  there  were 
disagreements.  Disagreements  indicate  macrocognitive  processes  that  are  not  mutually  exclusive. 
Definitions  for  some  macrocognitive  processes  may  need  to  be  refined  to  remove  ambiguity. 

Planning  tasks  can  also  trigger  decisions  to  take  action.  One  example  involves  flight  planning  by 
an  air  crew.  The  crew  may  have  a  discussion  regarding  the  weather  and  then  arrive  at  a  DTA  in 
the  form  of  deciding  to  change  the  route,  altitude,  amount  of  fuel,  or  even  the  destination.  These 
are  critical  decisions  as  a  wrong  decision  could  have  an  unfavorable  outcome.  Moreover,  these 
decisions  are  sometimes  modified  over  the  course  of  the  flight  as  new  information  is  gathered 
and  evaluated.  In  the  UAV  example  reported  in  the  results  section,  there  were  several  DTAs. 

At  this  time  our  working  hypothesis  is  the  more  a  task  or  problem  entails  dynamic  decision 
making  the  higher  the  number  of  DTAs  will  be  required.  We  plan  to  test  this  hypothesis  with 
analysis  of  additional  tasks  from  several  decision-making  and  planning  task  domains.  We 
anticipate  planning  tasks  will  entail  less  DTAs;  one  reason  is  that  planning  does  not  include  tasks 
that  require  dynamic  decisions  -  unless  the  task  entails  dynamic  planning/re-planning  while 
executing  the  task.  The  traditional  planning  process,  where  planning  for  the  task  is  performed 
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prior  to  execution,  is  somewhat  static  in  that  the  situation  does  not  change.  A  planning  task 
generally  entails  a  finite  set  of  decisions  in  contrast  to  the  many  emergent  decisions  that  arise  in 
dynamic  decision-making  situations.  The  state  of  the  world  changes  during  dynamic  decision¬ 
making  situations  but  not  during  planning  and  perhaps  this  is  the  biggest  difference  between 
planning  and  execution.  During  execution  tasks  decisions  are  made  in  real  time  and  must  be 
made  in  the  correct  moment  in  time,  whereas  time  is  not  really  a  factor  in  planning. 

CONCLUSIONS 

Decision  to  Take  Action  is  recommended  as  a  new  category  to  be  added  to  the  set  of 
macrocognitive  processes  included  in  the  Fiore,  Smith- Jentsch,  Salas,  Warner,  and  Letsky  (in 
press)  model  of  team  collaboration.  Deciding  to  take  action  is  viewed  as  both  a  macrocognitive 
process  and  a  product  of  team  collaboration.  We  maintain  that  decision  making  is  a  critical 
element  of  team  problem  solving  when  a  team  is  executing  a  task,  and  to  a  lesser  degree  when 
conducting  a  planning  task.  When  team  members  collaborate  to  solve  a  problem  they  make 
decisions  and  implement  those  decisions  as  part  of  performing  the  task.  Team  communications 
often  entail  asking  or  telling  a  team  member  to  perfonn  some  action  that  will  move  the  problem 
further  along  toward  completion.  This  contrasts  with  a  team  who  is  collaborating  on  a  planning 
task,  where  deciding  on  and  implementing  decisions  may  not  be  evident.  This  is  the  case  for 
problem-solving  domains  in  the  military,  government,  and  the  private  sectors.  For  example,  we 
predict  this  same  pattern  would  be  evident  for  a  surgical  team,  a  humanitarian  assistance 
scenario,  a  process  control  system  for  an  energy  plant,  and  a  variety  of  other  task  domains. 

Decision  Making  is  Part  of  Problem-Solving 

Many  critical  tasks  that  involve  team  problem  solving  include  decision  making;  that  is,  team 
members  take  action  in  addition  to  developing  new  knowledge  and  agreeing  on  a  final  solution. 
Actions  are  frequently  part  of  the  overall  information  gathering  process  and  have  diagnostic 
functions  (Orasanu  &  Connolly,  1993;  Orasanu  &  Fischer,  1997).  For  the  task  domains  discussed 
here,  a  constant  interplay  exists  between  sharing  information  -  to  develop  new  knowledge  and 
maintain  situation  awareness  -  and  deciding  on  actions,  and  implementing  these  actions, 
followed  by  monitoring  the  situation  and  continuing  to  build  new  knowledge  on  the  unfolding 
situation.  Execution  of  the  mission,  or  problem-solving  task,  would  come  to  a  screeching  halt 
without  this  continual,  iterative  cycle  of  developing  knowledge  of  the  situation  and  responding  to 
the  current  situation. 


ACKNOWLEDGEMENTS 

The  authors  wish  to  acknowledge  four  students  in  the  Graduate  School  of  Operational  and 
Information  Sciences,  at  the  Naval  Postgraduate  School,  who  coded  this  data  as  part  of  the 
research  conducted  for  their  master’s  thesis:  LCDR  Lenny  Samuel,  LT  Ken  Yates,  LT  Scott 
Place,  and  LT  Greg  Grubbs. 


18 


REFERENCES 


Air  Force  Research  Laboratory  Warfighter  Readiness  Division.  (2008).  Training  Exercise  09-1: 
Trainee  Preparation  Reader. 

Burke,  C.  S.  (2007).  Panel  Discussion  on  Macrocognition.  8th  International  Conference  on 
Naturalistic  Decision  Making.  Asilomar,  CA. 

Fiore,  S.  M.,  Smith-Jentsch,  K.  A.,  Salas,  E.,  Warner,  N.,  &  Letsky,  M.  (in  press).  Toward  an 
Understanding  of  Macrocognition  in  Teams:  Developing  and  Defining  Complex 
Collaborative  Processes  and  Products.  Theoretical  Issues  in  Ergonomic  Science. 

Fischer,  U.,  Orasanu,  J.,  &  Montalvo,  M.  (1993).  Efficient  decision  strategies  on  the  flight  deck. 
In  R.  Jensen  (Ed).  Proceedings  of  the  7th  Symposium  on  Aviation  Psychology  (pp.  238-243), 
Columbus,  OH:  Ohio  State  University. 

Hutchins,  S.  G.,  and  Kendall,  A.  (2008).  Patterns  of  Team  Collaboration  Employed  to  Solve 
Complex  Problems.  In  Proceedings  of  the  52nd  Annual  Meeting  of  the  Human  Factors  and 
Ergonomics  Society,  New  York,  September  22-26.  Santa  Monica,  CA. 

Hutchins,  S.  G.,  and  Kendall,  T.  (in  press).  The  Role  of  Cognition  in  Team  Collaboration  during 
Complex  Problem  Solving.  To  appear  in  K.  L.  Mosier  and  U.  M.  Fischer  (Eds.),  Informed  by 
Knowledge:  Expert  Performance  in  Complex  Situations.  Taylor  and  Francis. 

Katzenbach,  J.  R.,  &  Smith,  D.  K.  (1993).  Reengineering  the  corporation.  New  York:  Harper 
Collins  Publishers. 

Klein,  G.  A.  (1989).  Recognition-primed  decisions.  In  W.  R.  Rouse  (Ed.)  Advances  in  man- 
machine  systems  research  (Vol.  5,  pp.  47-92).  Greenwich,  CT:  JAI  Press. 

Klein,  G.  (1993).  Sources  of  error  in  naturalistic  decision  making  tasks.  In  Proceedings  of  the 
3  7th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society,  pp.  368-37 1 .  Santa 
Monica,  CA:  Human  Factors  and  Ergonomics  Society. 

Klein,  G.,  Phillips,  J.  K.,  Rail,  E.  L.,  &  Peluso,  D.  A.  (2007).  A  data-frame  theory  of 

sensemaking.  In  R.  Hoffman  (Ed.),  Expertise  out  of  context  (pp.  1 13-155).  New  York,  NY: 
Lawrence  Erlbaum  Associates. 

Klein,  G.,  Ross,  K.  G.,  Moon,  B.  M.,  Klein,  D.  E.,  Hoffman,  R.  R.,  &  Hollnagel,  E.  (2003). 
Macrocognition.  IEEE  Intelligent  Systems,  18(3),  81-85. 

Klein,  G.,  Snowdon,  D.,  &  Pin,  C.  L.  (2007).  Anticipatory  Thinking.  Proceedings  of  the  8th 
International  NDM  Conference.  (Ed.s  K.  Mosier  &  U.  Fischer),  Pacific  Grove,  CA,  June 
2007. 

Letsky,  M.,  &  Warner,  N.  (2008).  Macrocognition  in  teams.  In  M.  Letsky,  N.  Warner,  S.  M. 
Fiore,  &  C.  Smith  (Eds.).  Macrocognition  in  Teams  (pp.  1-13).  London:  Ashgate. 

Letsky,  M.,  Warner,  N.,  Fiore,  S.  M.,  Rosen,  M.,  &  Salas,  E.  (2007,  June).  Macrocognition  in 
complex  team  problem  solving.  In  Proceedings  of  the  12th  International  Command  and 
Control  Research  &  Technology  Symposium,  Newport,  RI. 

Memorandum  of  understanding  between  NORAD  and  the  FAA.  (1987).  HQ  North  American 
Aerospace  Defense  Command  Regulation  55-18.  Peterson  Air  Force  Base,  Colorado  80914- 
5001  23.  November,  1987. 

Montgomery,  H.  (1983).  Decision  rules  and  the  search  for  a  dominance  structure:  Towards  a 
process  mode  of  decision  making.  In  P.  Humphreys,  O.  Svenson,  &  A.  Vari  (Eds.),  Analyzing 
and  aiding  decision  processes  (pp.  343-369).  Amsterdam:  North-Holland. 


19 


Montgomery,  H.  (1989).  From  cognition  to  action:  The  search  for  dominance  in  decision 
making.  In  H.  Montgomery,  &  O.  Svenson.  (Eds.),  Process  and  structure  in  human  decision 
making  (pp.  23-49).  Chichester,  England:  Wiley. 

Mosier,  K.  L.  (1991).  Expert  decision  making  strategies.  In  R.  Jensen  (ed.),  Proceedings  of  the  Sixth 
International  Symposium  on  Aviation  Psychology ?  (pp.  266-271).  Columbus,  OH:  Ohio  State 
University. 

Nosek,  J.  T.  (2004).  Group  cognition  as  a  basis  for  supporting  group  knowledge  creation  and 
sharing.  Journal  of  Knowledge  Management,  5(4),  3-18. 

Orasanu,  J.  M.  (1994).  Shared  problem  models  and  flight  crew  performance.  N.  Johnston,  N. 
McDonald,  &  R.  Fuller  (Eds.),  Aviation  psychology  in  practice  (pp.  1-22).  Aldershot, 
England:  Ashgate. 

Orasanu,  J.,  &  Connolly,  T.  (1993).  The  reinvention  of  decision  making.  In  G.  A.  Klein,  J. 
Orasanu,  R.  Calderwood,  &  C.  E.  Zsambok  (Eds.),  Decision  making  in  action:  Models  and 
methods  (pp.  3-20).  Norwood,  NJ:  Ablex. 

Raytheon  Company  (2008).  JADOCS  Manual.  JADOCS  Overview.  Alexandria,  VA 

Salas,  E.,  &  Fiore,  S.  M.  (2004).  Team  cognition:  Understanding  the  factors  that  drive  process 
and  performance.  Washington,  DC:  American  Psychological  Association. 

Thomas,  J.,  and  Hersen,  M.  (2003).  Understanding  Research  in  Clinical  and  Counseling 
Psychology.  Lawrence  Erlbaum:  Lawrence  Erlbaum  Associates. 

Warner,  N.,  &  Letsky,  M.,  (2008).  Empirical  model  of  team  collaboration  focus  on  macro¬ 
cognition.  In  M.  Letsky,  N.  Warner,  S.  Fiore,  C.  Smith,  (Eds.)  Macrocognition  in  teams  (pp. 
15-33).  London:  Ashgate. 

Warner,  N.,  Letsky,  M.,  &  Cowen,  M.  (2005).  Cognitive  model  of  team  collaboration:  Macro- 
cognitive  focus.  In  Proceedings  of  the  49th  Human  Factors  and  Ergonomics  Society  Annual 
Meeting  (pp.  269-273).  Santa  Monica,  CA:  Human  Factors  and  Ergonomics  Society. 

Wood,  D.  J.,  &  Gray,  B.  (1991).  Toward  a  comprehensive  theory  of  collaboration.  Journal  of 
Applied  Behavioral  Science,  27,  149-162. 


20 


